feat(compiler): Add grpc support for Swift - #3776
Conversation
Schemas with services now emit a <Service>Grpc.swift companion beside the Swift model. Each service gets Fory-backed async and NIO providers plus an async client; request and response bytes ride a private GRPCPayload wrapper that serializes through the schema module's Fory instance.
Before writing Swift output, check that no two schemas or services claim the same file path or top-level symbol. A service named after a generated type, or a duplicate service, now fails fast with a clear message instead of emitting Swift that will not compile.
Exercise the Swift companion across the four streaming shapes, keyword-escaped methods, imported request and response types, the default package, both IDL frontends, and the collision preflight, so the emitter and its symbol names stay pinned.
Generate a two-package schema, then swift build and run a SwiftPM package on grpc-swift and local Fory that hosts the generated provider and round-trips all four streaming shapes across the import boundary. Skipped when swift is absent.
Wire Swift into the shared gRPC generation step so the interop schemas emit Swift companions alongside the other targets.
Add a Swift gRPC guide covering dependencies, server and client usage, streaming, and troubleshooting, link it from the Swift guide index, and note the Swift companion in the compiler guide and agent rules.
Break the streaming handler closures across lines so generated companions stay under the swiftlint line-length limit even with long package-qualified names.
Put handler braces on the declaration line, give each async parameter its own aligned line, name the unwrapped stream value, and scope a type_name disable around the package-prefixed symbols so swiftlint reports no violations.
Schemas that share a top-level package component make the model generator emit a duplicate root enum, which the Swift compiler rejects in one module. Pin it with a strict xfail fixture and a docs note pointing at disjoint packages.
The Swift Fory instance is single-threaded, but gRPC drives the marshaller from many threads at once, so sharing one instance races. Build one Fory per thread from the module config and registrations, and fire 200 parallel calls in the fixture to exercise it.
Record that the generated client is async only and that interceptors are not emitted, both because grpc-swift types them on the internal Fory wrapper, and describe the per-thread marshalling.
Name the wire wrapper per service so it is reachable, then drive it from 2000 parallel threads under ThreadSanitizer, asserting no data race and that the per-thread Fory stays wire-compatible with the module's shared instance. Against a shared instance TSan flags a race in the type resolver.
The ThreadSanitizer build adds about three minutes and is environment sensitive, so keep it opt-in for a sanitizer or nightly job while the functional fixtures still run on every swift-capable run.
An rpc whose Swift name is handle, serviceName, channel, or defaultCallOptions would clash with a member the generated provider or client inherits, so fail codegen with a clear message. Cover the reserved names and nested plus imported request and response payloads.
# Conflicts: # compiler/fory_compiler/tests/test_service_codegen.py # docs/compiler/compiler-guide.md # integration_tests/grpc_tests/generate_grpc.py
|
@yash-agarwa-l Please git merge apache/main first to address the conflicts |
|
I already did that, so I'm not sure why it's still showing up. Let me mark it as ready for review and double-check. |
The SwiftPM build-and-run round-trip and the ThreadSanitizer marshaller test need the Swift toolchain, so move them out of the compiler pytest suite (where they only skipped) into a SwiftPM package under integration_tests/grpc_tests/swift. The common-root package limitation stays pinned as a build-free generation check.
Mark the generated message wrapper @unchecked Sendable, a transient single-owner carrier that is serialized synchronously and never shared, so grpc-swift's async APIs accept it under Swift 6 strict concurrency, and make its value immutable. Wrap the service descriptor method list to stay within the line limit. Document that companions compile in Swift 5 language mode until generated models are Sendable.
Point the shared generator at the Swift package's generated sources and run the relocated marshaller round-trip and concurrency tests with `swift test` when the toolchain is present, so they execute outside the JVM-driven suite
|
Hi @chaokunyang, small update and one thing I'd like your call on. The Swift gRPC codegen is done and Swift↔Swift round-trips fine, but Java↔Swift interop fails. After digging in, it looks like a gap in the Fory Swift core serializer, not the gRPC code. When refTracking is on, Swift writes If that sounds right to you, I'd like to do the same for Swift. One thing to flag: it changes tracking-mode output for all Swift structs, not just gRPC. Tracking is opt-in and off by default, so the impact should be small, but it's a core wire-format change, so I wanted to ask first. No gRPC module changes; the contract stays refTracking:true. Two questions:
Since it's a wire-format change, I'm happy to do whatever works best for you here, a separate issue or PR, or just keep it on this PR. Let me know. |
|
When ref tracking is enabled, java/python will still push an id -1 to ref reader, and when inner struct invoke /** Stores {@code object} under an already reserved read ref id. */
@Override
public void setReadRef(int id, Object object) {
if (id >= 0) {
readObjects.set(id, object);
}
}There are some code invoking if (refMode != RefMode.NULL_ONLY || buffer.readByte() != Fory.NULL_FLAG) {
refReader.preserveRefId(-1);
return readContext.readNonRef(fieldInfo.typeInfo);
} |
| return symbols | ||
|
|
||
| def _swift_grpc_method_name(self, method: RpcMethod) -> str: | ||
| return self.safe_member_name(method.name) |
There was a problem hiding this comment.
Reject or rename an underscore-only RPC method
FDL accepts an RPC named _, but the generated member remains _, producing declarations and references such as static let _, func _, and Methods._ that Swift cannot use as identifiers. Please generate a referencable name or reject this RPC name during preflight.
| "// loop, so the payload must be Sendable. The carrier itself only stores that", | ||
| "// payload, so @unchecked covers the wrapper while Value carries the guarantee.", | ||
| ( | ||
| f"struct {base}Message<Value: Serializer & Sendable>: GRPCPayload," |
There was a problem hiding this comment.
Align the Sendable constraint with generated model types
This constraint applies to every request and response wrapper, but generated public structs, enums, unions, and classes do not conform to Sendable. Swift 6 therefore rejects even unary providers and clients when Message<Request/Response> is instantiated, not only client-streaming and bidi as the guide states. Please add a safe Sendable strategy plus Swift 6 build coverage, or state that the entire companion currently requires Swift 5 mode.
| ```swift | ||
| // Package.swift | ||
| dependencies: [ | ||
| .package(url: "https://github.com/apache/fory.git", from: "1.2.0"), |
There was a problem hiding this comment.
Require a Fory version that contains this runtime support
from: "1.2.0" allows versions whose Swift Serializer lacks Target, and even 1.6.1 lacks this PR's tracked-value root ref-slot fix required for Java/Swift interoperability. Please use the first release containing both the compiler and runtime changes, or follow the existing $version convention.
| final class RoundTripTests: XCTestCase { | ||
| func testInProcessAllStreamingModes() async throws { | ||
| let group = MultiThreadedEventLoopGroup(numberOfThreads: 1) | ||
| defer { try? group.syncShutdownGracefully() } |
There was a problem hiding this comment.
Use async cleanup in the async test
The current Java/Swift CI job reports async-context warnings for this blocking shutdown, lines 118/124, and main.swift:456; Swift 6 upgrades at least one to an error. Please use awaited get() or structured async lifecycle cleanup while preserving failure-path cleanup. Swift changes are required to compile without warnings.
| - name: Run Swift gRPC package tests | ||
| run: | | ||
| cd integration_tests/grpc_tests/swift/interop | ||
| swift test |
There was a problem hiding this comment.
Run the concurrency regression under ThreadSanitizer
This runs only plain swift test. The sole --sanitize=thread command is behind FORY_SWIFT_TSAN=1 in another script that this workflow never invokes, so the test and PR description's TSan safety claim is not exercised by CI. Please add a sanitized invocation, preferably filtered to the marshaller concurrency test.
| fi | ||
| # Swift toolchain tests (generated marshaller round-trip and concurrency). These | ||
| # need the Swift toolchain rather than the JVM, so they run in their own package. | ||
| if command -v swift >/dev/null 2>&1 && [ -d "${SCRIPT_DIR}/swift/interop" ]; then |
There was a problem hiding this comment.
Honor the selected test classes for Swift work
This block ignores TEST_CLASSES, so ./run_tests.sh GoGrpcTest still resolves and runs the Swift package whenever Swift is installed. Please put both the Swift package tests and the SwiftGrpcTest build behind has_test_class "SwiftGrpcTest", while retaining the tool-availability check before invoking Swift.
fory-compiler declares requires-python >=3.8, but cli.py and the Swift gRPC service generator used PEP 585 builtin generics and PEP 604 unions in function signatures. Those evaluate at import time, so foryc raised TypeError on startup under 3.8 and 3.9. Use typing.List/Dict/Set/Tuple/Optional, matching the other ten service generators.
The module used PEP 585 builtin generics in annotations, which Python 3.8 evaluates at import time. Match the compatible syntax now used by cli.py and the Swift gRPC service generator.
The FDL, proto, and FlatBuffers parsers annotated a return type with a PEP 585 builtin generic. Importing the CLI loads all three eagerly, so foryc raised TypeError on 3.8 even after the CLI itself was fixed.
fory-compiler declares requires-python >=3.8, but the compiler job runs 3.11, so annotation syntax newer than the declared floor reached main unnoticed. Import the CLI on the oldest supported interpreters, which covers every module it loads eagerly.
The thread-local key held only the generated textual type path, so every default-package schema produced org.apache.fory.grpc.ForyModule. Two generated targets in separate Swift modules shared that key, and the second wrapper reused the first module's Fory without its own registrations. Build the key from String(reflecting:), which is module-qualified at runtime.
Enum-style packages nest their types under a namespace enum, but that enum is itself a file-scope declaration. Preflight recorded no symbol for it, so two schemas sharing a top-level package component passed and Swift then rejected the duplicate declaration. Record the namespace owner so generation fails first.
The symbol collector returned a set, so a name declared twice by one schema collapsed to a single entry and preflight reported no collision. Report every declaration with the Swift scope that contains it, keeping duplicates, so the generated helper against a same-named message, flattened helper names, and names that normalize to the same identifier all fail before generation.
| XCTAssertEqual(fromMarshaller, probe) | ||
| } | ||
|
|
||
| func testSeparateModulesKeepOwnRuntimeOnOneThread() throws { |
There was a problem hiding this comment.
Make the runtime-key regression test reproduce the old collision
GrpcFdl and GrpcFbs are both compiled into the single ForyGrpcGenerated Swift target, and the old textual keys were already different (GrpcFdl.ForyModule versus GrpcFbs.ForyModule). This test therefore passes even if the String(reflecting:) fix is removed. Please exercise two Swift targets/modules that expose the same textual helper path, ideally default-package ForyModule, so the test fails on the old implementation.
| hooks would expose that wrapper. Use a custom channel or server configuration for | ||
| cross-cutting concerns instead. | ||
|
|
||
| Swift models put each package under a nested `enum` namespace, so two schemas that |
There was a problem hiding this comment.
Update the namespace limitation for the new preflight
The compiler now rejects schemas with the same root namespace before writing either file, so this section no longer describes the actual failure point. The separate-module workaround also requires separate generation invocations because one invocation runs the shared collision preflight before module assignment is known. Please document the preflight failure and that requirement.
There was a problem hiding this comment.
The preflight wording is closer, but the separate-invocation workaround does not work for the service-import example above. validate_swift_files() recursively collects imports, so compiling demo.greeter still includes demo.shared in the graph and rejects the duplicate Demo even if the shared schema was generated in another invocation. Separate invocations only help unrelated schemas; an import graph currently needs disjoint top-level packages. Please narrow the workaround accordingly.
… services The wrapper is emitted only when a service declares methods, but preflight registered it unconditionally, so an empty service rejected a schema type of the same name that never collided. Gate the symbol on the service having methods and assert declared symbols match the emitted ones.
The namespace section still described an invalid redeclaration at build time, but the compiler now rejects a shared root namespace before writing either file, and the separate-module workaround needs one invocation per module because a single invocation preflights every schema together. The language mode section also predated the Sendable payload constraint, which applies to all call shapes rather than only the streaming client.
Both marshaller regression tests asserted coverage they did not have. testSeparateModulesKeepOwnRuntimeOnOneThread claimed to guard the module-scoped ForyRuntime key from d2a47f2, but it used GrpcFdl and GrpcFbs, which compile into one target and carry packages, so their keys were already distinct under the old scheme and it stayed green when that scheme was restored. Add two package-less schemas generated into their own SwiftPM targets: both emit a bare ForyModule, so their generated key expressions are identical and only runtime module qualification separates them. Each needs its own foryc invocation, since one invocation preflights its schemas together and rejects the duplicate declaration. Against the old "org.apache.fory.grpc.{module}" scheme the new test fails with "Type not registered: DefaultPackageTwoRequest is not registered". The old test is renamed to testPackagedSchemasKeepOwnRuntimeOnOneThread, which is what it actually covers. The concurrency test claimed to run under ThreadSanitizer in CI, which ran only plain swift test. The one --sanitize=thread command sat behind FORY_SWIFT_TSAN in run_tests.sh, which the workflow never invokes and nothing sets. That command was also broken on macOS rather than merely unreachable: SwiftPM launches XCTest bundles through the platform-signed swiftpm-xctest-helper and xctest, and dyld refuses to load the sanitizer runtime into them, while SIP strips DYLD_INSERT_LIBRARIES. A swift-testing runner is a plain executable and has no such restriction, so the suite moves to swift-testing and CI runs it with --sanitize=thread --disable-xctest. Reverting the per-thread Fory to a shared instance makes TSan report 70 races in TypeResolver.finalizeTypeMeta and exit 1, so a race fails the step without TSAN_OPTIONS. The step greps for the suite result so it cannot decay into a zero-test no-op. Delete FORY_SWIFT_TSAN and its run_tests.sh block: nothing set it, its command did not work, and CI now owns the sanitized run.
…eholder from: "1.2.0" advertised a floor that permits releases whose Swift Serializer lacks Target, and that predate the tracked-value root ref-slot fix needed for Java/Swift interop. Naming the first release containing both is not possible yet: df15e7c is in no tag and is not an ancestor of v1.7.0-rc1, so no cut release carries it and any concrete number would be a guess. Use the existing convention instead and match docs/object-serialization/swift/index.md verbatim with exact: "$version". ci/release.py leaves the placeholder alone because VERSION_PATTERN matches only numeric versions, and the adjacent grpc-swift line does not match _is_release_doc_line, so a Fory release cannot rewrite grpc-swift's version.
The in-process round-trip test and the interop client shut down their gRPC and NIO resources from `defer` blocks using syncShutdownGracefully() and EventLoopFuture.wait(). Both are unavailable from asynchronous contexts, so the Java/Swift CI job reported four warnings and the Swift 6 language mode turns the syncShutdownGracefully() one into an error. wait() on an event loop thread can also deadlock, which is why NIO marks it unavailable rather than merely noisy. `defer` cannot await, so the test collects an async teardown step per resource and unwinds them in reverse on both exit paths, and the client closes its channel explicitly on the success and failure paths. Teardown behaviour is unchanged: verified by instrumenting each step and observing channel, server, then group run when the body succeeds and when it throws. Our sources now build with zero warnings; the remaining ones come from third-party checkouts.
Bring the Swift gRPC branch up to date with main, which has moved 28 commits ahead including the 1.6.1 release version bump.
…workaround The Swift package tests were gated only on `command -v swift`, so ./run_tests.sh GoGrpcTest resolved and ran them whenever a toolchain was installed, and the SwiftGrpcTest release build sat in a second block. Both now share one has_test_class "SwiftGrpcTest" guard, keeping the tool-availability check before Swift is invoked. The namespace section also told users that generating into separate Swift modules with one foryc invocation each works around a shared root namespace. That only holds for unrelated schemas. validate_swift_files builds its graph through collect_schema_graph, which walks imports recursively, so compiling demo.greeter still includes demo.shared and rejects the duplicate Demo even when the shared schema was generated in another invocation. An import graph needs disjoint top-level packages, and the section now says so.
safe_member_name routes the rpc name through to_camel_case, which strips underscores, so an all-underscore name normalizes to empty and safe_identifier returns its "_" fallback. rpc _, __, and ___ all reached it and emitted func _, static let _, and Methods._. swiftc rejects func _ outright, while static let _ parses as a wildcard binding and fails only where Methods._ is referenced. Reject instead of renaming: an invented name would be arbitrary and could collide on its own. The check joins the existing method-name preflight, which the CLI reaches before writing any file, so it exits with a clear error and leaves no partial output. That preflight now covers unusable names, reserved members, and duplicates, so it is named _check_swift_grpc_method_names. Document both rpc-name rules, including the reserved-member list that was already enforced but never written down.
swift_declared_symbols() iterated only the schema's top-level enums, unions, and
messages, so nested declarations never reached the collision preflight. A schema
such as message Parent { message my_type {} message MyType {} } passed and then
emitted two public struct MyType inside Parent. The failure is semantic rather
than syntactic, so swiftc reports invalid redeclaration only at type-check.
Walk nested messages, enums, and unions recursively and key each declaration by
its parent Swift scope. Enums and unions stop the walk because only Message
carries nested collections. Nested names keep the bare normalized form, so the
flatten namespace prefix still applies to the top-level owner alone.
test_swift_nested_normalized_name_collision_fails_preflight described package
namespace nesting rather than an IDL nested type, so it is renamed to
..._packaged_..., and real nested cases now have their own tests, including a
deeper level and same-named types under distinct parents.
…d code The generated models declare Equatable but not Sendable, and the gRPC wire wrapper constrains its payload to Sendable, so Swift 6 strict concurrency rejects every call shape rather than only the streaming ones. Say so in the model documentation as well, because the limit applies to any Swift 6 code that moves a generated value across an isolation boundary, not only to gRPC users. Drop the claim that Swift 6 support follows once models are Sendable: messages using ref or weak fields generate classes with mutable stored properties, which cannot declare Sendable at all.
Why?
Swift users can generate Fory model types today, but schemas that define services
do not produce Swift gRPC companions. This leaves Swift out of the existing
--grpcworkflow used by the other supported service-generation targets.
What does this PR do?
metadata descriptors, an
EventLoopFutureprovider, an async/await provider, anasync client, and Fory-backed request/response stream adapters, targeting grpc-swift 1.x.
internal
GRPCPayloadwrapper. Because the SwiftForyruntime is single-threaded,the wrapper builds one
Foryper thread from the schema module's own configurationand registrations, so concurrent RPCs are race-free (verified under ThreadSanitizer).
EventLoopFutureclient and interceptor hooks areomitted because their generated types would expose the internal wrapper.
collisions, and reserves inherited provider/client member names (
handle,serviceName,channel,defaultCallOptions) so a clashing rpc fails codegenwith a clear message.
identifier escaping, imported and nested service types, the default package, the
protobuf and FlatBuffers frontends, collision handling, a SwiftPM build-and-run
fixture, and a marshaller concurrency test under ThreadSanitizer (gated behind
FORY_SWIFT_TSAN).shared-top-level-package limitation, troubleshooting, and compiler guide updates.
Draft: Java<->Swift cross-language interop tests (all four modes, all three IDL
frontends) are in progress and will be added before this PR is marked ready for review.
Related issues
#3266
#3370
AI Contribution Checklist
yesyes, I included a completed AI Contribution Checklist in this PR description and the requiredAI Usage Disclosure.yes, my PR description includes the requiredai_reviewsummary and screenshot evidence of the final clean AI review results from both fresh reviewers on the current PR diff or current HEAD after the latest code changes.Does this PR introduce any user-facing change?
foryc --swift_out=... --grpcis used.change the Fory binary protocol.
Benchmark
Not applicable.